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ABSTRACT 

A telerobotic control station is described. In it, a machine vision system measures the position, 
orientation, and configuration of a user’s hand. A robotic manipulator mirrors the status of the 
hand. This concept has two benefits: control actions are intuitive and easily learned, and the 
workstation requires little volume or mass. 

INTRODUCTION 

Telerobotic systems have great potential for making space assets more flexible and productive. In 
particular, astronauts might use telerobotic systems to reduce the need for extravehicular activity or 
to handle hazardous tasks. The design of a telerobotic control station for use in manned spacecraft 
must meet several constraints. The control station interface should be easy to use and to learn. 
Mass and volume are strongly constrained in spacecraft, so the control station's mass and volume 
must be minimized. Repair or replacement of spacecraft equipment can be costly or impractical, so 
reliability must be high. 

The requirement to be easy to use and learn is often unmet due to configuration differences 
between the manipulator and the control interface. Many robot manipulators bear a superficial 
resemblance to the human hand. They have grippers which crudely approximate the human thumb 
and fingers, and wrists which pitch and roll as the human wrist does. These manipulators are 
often attached to robot arms which approximate the human elbow and, to a lesser extent, the 
shoulder. In contrast, human control interfaces to robot arms and manipulators have typically used 
joysticks. The joystick is not a particularly natural interface, whether for robots or for other 
equipment. Substantial training and practice are required to reach moderate skill levels, and 
relatively few individuals have enough talent to efficiently use the device in tasks which demand 
precision. (The late Dr. Judith Resnick was noted for her exceptional skill at manipulating the 
Space Shuttle's robot arm via the joystick-based control station.) 

Besides being a poor interface, a joystick control station takes up scarce volume and wall area. 

This space is unavailable for other needs even when the control station is not in use. 

Joysticks are mechanical devices which must endure friction and stress. To be reliable, they must 
be robustly built. This requires substantial mass, expensive high-strength material, or both. 

Two existing possible substitutes for joystick control are the DataGlove 1 and the Sensor Frame 2 . 
The DataGlove is a special glove which measures the position, orientation, and configuration of the 
user's hand. The Sensor Frame is a set of infrared light sources and detectors aiTayed around the 
screen of a video display; the user's fingers interrupt beams of light, thus revealing the fingers' 
positions. Both DataGlove and Sensor Frame are expensive and require special hardware. 

This paper describes a vision-based control station which is intended to avoid the disadvantages of 
joysticks, and to do so with lower cost and less dedicated hardware than the DataGlove and Sensor 
Frame. It provides a natural- seeming interface, requires tittle mass and volume, and has no 
moving mechanical parts. A simple implementation of the control station concept has been 
assembled and tested. The first section below describes the physical configuration of the control 
station, its computational support, and its connection to the robot. The second section describes 
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the algorithm for image interpretation and robot control. The third describes two alternatives for 
mapping hand positions to manipulator actions and explains the rationale used for selection 
between them. The fourth section briefly presents the results of initial tests with the control 
station. Finally, areas for further development are discussed. 

INTERFACE CONFIGURATION 

The control station consists of a video camera, a known visual background, and a specially marked 
glove which the user wears. In a fully developed control station, the glove should be unnecessary. 
(The control station also includes a video display, which provides feedback about the robot’s 
actions and environment. This paper does not address the subject of feedback.) The user places 
his or her gloved hand in the video camera's field of view. An image processing computer detects 
the hand and measures its position, orientation, and configuration. These measurements are 
mapped into a set of manipulator commands which are sent to the robot. Thus, the user can direct 
the robot to flex its wrist by tilting his or her hand, or can direct the manipulator to close its gripper 
by bringing his thumb and forefinger together. 

The current hardware configuration of the system is shown in Figure 1. The user's hand motions, 
observed by the video camera, determine the actions of a robot gripper in another room. The Sun 
workstation handles most of the non-image processing load; the Explorer workstation only 
transfers commands from the Ethernet to the robot. Not shown in the figure are a television 
camera and a monitor which allow the user to view the robot. 
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Figure 1. Current lab configuration of control station. 

The glove is marked as shown in Figure 2. Viewed against a dark background, the light glove is 
easy to detect by thresholding. The dark rings on the thumb and index finger are detected by 
morphological image operations, allowing the hand's roll position and its grip width (distance 
between thumb and index finger) to be measured. 
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ALGORITHM 

The robot manipulator has several parameters, e.g. pitch or grip width, each having a finite number 
of states. Each parameter is independent, that is, the state of one parameter has no effect on the 
state of another. The purpose of the program is to control the state transitions of each manipulator 
parameter and thereby to control the manipulator itself. 

The control station algorithm is presented below as Algorithm 1. The algorithm's outer structure is 
an eternal loop. In each pass through the main loop, the first step is for the computer to examine 
an image, determine whether a glove is present, and if so, measure its position and orientation 
parameters. Each parameter is discretized, with each discrete value mapping to one state of the 
corresponding manipulator parameter. If the indicated state does not match the current 
manipulator state, then the program issues a command to change the robot’s state. To avoid 
spurious state changes, the algorithm only sends a command to the robot if the same change is 
indicated in two consecutive images. If the glove is removed from the field of view, the robot 
maintains its current state. 

Initialize the current-state for each manipulator parameter. 

While (True) Do: 

Acquire image. 

Threshold glove from background. 

If glove in view: 

Measure glove position/orientation parameters. 

For each glove parameter: 

Quantize. 

Map to indicated state for manipulator parameter. 

If indicated-state matches current-state for the parameter: 

Do nothing. 

Else: 

If indicated-state matches previous-indicated-state: 

Set current-state to indicated-state. 

Send appropriate command to robot. 

Else: 

Set previous-indicated-state to indicated-state. 

Algorithm 1. Hand position interpreter 
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MAPPING STRATEGIES 


There are two straightforward ways to interpret glove position: either as a direct indicator of 
manipulator position, or as an indicator of the manipulator's rate and direction of motion. In 
position-to-position mapping, the robot hand moves to the same orientation as the user's hand. In 
position-to-rate mapping, the user moves his hand from the neutral position to a position which 
indicates motion in some direction or about some axis, e.g. to increase the wrist pitch, the user 
flexes his wrist upward. The robot wrist rotates upward continuously until reaching its limit or 
until the user returns his wrist to a neutral or downward position. 

Position-to-rate mapping was selected because it allows the robot wrist to roll through its full range 
of motion, i.e. infinite in either direction, as long as the user's hand is tilted to the side. An 
implementation with position-to-position mapping would be unable to roll the wrist beyond the 
finite limits of human wrist motion. The position-to-position mapping may be easier to learn and 
use than the position-to-rate mapping, but that hypothesis has not been tested in this work. 

Manipulator parameters currently supported are wrist pitch and gripper width. Wrist pitch is 
controlled as described above. Gripper width is determined by distance between the tips of the 
thumb and the index finger: small distance closes the gripper, large distance opens it, and neutral 
distance stops the gripper at the current width. 

RESULTS 

The system currently lets the user control the robot manipulator in two parameters with fair 
reliability (estimated at about 80% for each command motion). Response time is about 1.5 
seconds, due to the rather slow image processing software, the specification that two consecutive 
images must agree on a command, and a slow data link to the robot. Removing the hand from the 
field of view and later returning the hand to the field of view usually does not confuse the interface 
software. It is expected that the system can be speeded up and made more reliable with moderate 
effort. Development of the system so far has required roughly 30 man-hours. 

DISCUSSION 

The equipment required for the control station is limited to a marked glove, a video camera, and 
computers. A video camera and computers will probably be present on any manned spaceflight, so 
the only unique equipment required is the glove. Future efforts may remove the requirement for a 
glove. 

Further development of the vision-based control station is warranted. Logical extensions are to 
increase the number of parameters that are supported, to increase the speed, and to enable 
operations with a bare hand rather than a glove. Increasing speed and number of parameters are 
fairly straightforward. Operating without a glove can probably be accomplished by subtracting 
each image from the known background image; this would reduce dependence on visual contrast 
between the hand and the background. 

A significant challenge for the vision system is to distinguish the positions of all five fingers, 
should that be necessary for future dextrous manipulators. A reasonable approach may be to use a 
color camera and to require the user to wear different colored rings on each finger. This avoids the 
inconvenience of a glove, yet eases the task of the vision system. 
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SUMMARY 


A vision-based telerobotic control interface has been described. The interface requires little or no 
special equipment, which is an advantage for space applications. It provides a natural 
correspondence between user actions and robot motions. 

The interface has been partially implemented with little effort. The current implementation has been 
successfully tested. However, the interface requires further development to improve reliability and 
speed. 


'DataGlove is a product of VPL Research, Redwood City, CA. 

2 Sensor Frame is a product of Sensor Frame Corporation, Pittsburgh, PA. 
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